Many types of computing systems and applications generate vast amounts of data pertaining to or resulting from the operation of that computing system or application. These vast amounts of data are stored into collected locations, such as log files/records, which can then be reviewed at a later time period if there is a need to analyze the behavior or operation of the system or application.
Server administrators and application administrators can benefit by learning about and analyzing the contents of the system log records. However, it can be a very challenging task to collect and analyze these records. There are many reasons for these challenges.
One significant issue pertains to the fact that many modern organizations possess a very large number of computing systems, each having numerous applications that run on those computing systems. It can be very difficult in a large system to configure, collect, and analyze log records given the large number of disparate systems and applications that run on those computing devices. Furthermore, some of those applications may actually run on and across multiple computing systems, making the task of coordinating log configuration and collection even more problematic.
Conventional log analytics tools provide rudimentary abilities to collect and analyze log records. However, conventional systems cannot efficiently scale when posed with the problem of massive systems involving large numbers of computing systems having large numbers of applications running on those systems. This is because conventional systems often work on a per-host basis, where set-up and configuration activities need to be performed each and every time a new host is added or newly configured in the system, or even where new log collection/configuration activities need to be performed for existing hosts. This approach is highly inefficient given the extensive number of hosts that exist in modern systems. Furthermore, the conventional approaches, particularly on-premise solutions, also fail to adequately permit sharing of resources and analysis components. This causes significant and excessive amounts of redundant processing and resource usage.
In addition, to know what data needs to be retrieved from a given log file, there should be a way to identify which, if any, of the log files have actually changed enough to warrant retrieval of its log data. This is because the log analytics system only needs to retrieve data from log files that have included new content since the last time period when the log data was retrieved for processing and analysis.
Known approaches to identify changed log files suffer from severe inefficiencies. The most common approach is to scan through all of the log files one-by-one in a given host system to identify whether any of the files have changed. The timestamp and/or size of the files can be inspected to see if there have been any changes since the last time period at which log data was retrieved. Alternatively, the checksum and/or MD5 value for the file can be checked to verify the existence of any changed files.
The problem with these approaches is that the environment may include a large number of host systems, where each of the host systems may contain a very large number of files pertaining to a large number of different targets that need to be checked for changes. This means that there may be potentially thousands or millions of different files that need to be inspected for possible changes. Moreover, out the large number of files to review, it is likely that at any given moment in time, only a small number of those files have actually undergone a change that would necessitate retrieval of its log data. As such, the approach of iterating through each and every one of the files to check for changes would likely consume a significant amount of computing resources and time for a very small payoff, resulting in a very lengthy, expensive, and inefficient sequence of actions.
Some embodiments provide an improved approach for identifying log files that have undergone a change in status that would require retrieve of its log data. Other additional objects, features, and advantages of the invention are described in the detailed description, figures, and claims.